Strobe image composition method, apparatus, computer, and program product

ABSTRACT

There are disclosed a method, apparatus, computer, and program product for generating a strobe composite image from a plurality of frames of a moving image. A first frame is selected from the plurality of frames of the moving image. A plurality of second frames relating to the first frame are determined. Then, a strobe composite image is generated by superposing the plurality of second frames in accordance with a superposing manner.

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is based upon and claims the benefit of priorityfrom the prior Japanese Patent Applications No. 2002-288014, filed Sep.30, 2002; and No. 2003-068968, filed Mar. 13, 2003, the entire contentsof both of which are incorporated herein by reference.

BACKGROUND OF THE INVENTION

[0002] 1. Field of the Invention

[0003] The present invention relates to a method, apparatus, computer,and program product for generating a strobe composite image bysuperposing time-serially obtained object images.

[0004] 2. Description of the Related Art

[0005] An image processing technique for generating and displaying acomposite image by extracting only an object portion from a moving imageand superposing it on another image is effective to generate a strobecomposite image by superposing object images sensed at a plurality oftimes. In this case, if a user interface that allows the user tosuperpose a plurality of object images is not available, the user cannoteasily generate a strobe composite image, and must spend much time inoperations.

[0006] Conventionally, a technique for superposing a plurality of imagecomponents called layers each having a transparent portion has been usedto generate a strobe composite image by superposing a plurality ofobject images (for example, see reference “Adobe Premier 6.0 Manual,Chapter 7, pp. 281-294”). Initially, only object image regions areextracted from a moving image by an arbitrary method. In this process, achroma-key method that senses motion of an object in front of a uniformcolor background, and extracts object images by exploiting colorcomponent differences can be adopted. Then, the extracted object imageregions are copied to form a plurality of layers using a layer functionof image processing software, and a strobe composite image can beobtained by superposing these layers.

[0007] However, in order to create a strobe composite image by theconventional technique, the user must manually create layerscorresponding to frames, and must manually designate the superposingmanner as a hierarchy upon creating layers or must manually adjust thehierarchy of layers after the layers are created. Also, in theconventional technique, the user must execute strobe composition bycapturing an image which is to undergo strobe composition, and saving itin a file. For example, if it is impossible to photograph an object infront of a uniform background like in sport games, the chroma-key methodcannot be used to extract an object portion from an image. Therefore, amanual extraction process is required, thus taking much time until astrobe composite image is created.

[0008] It is often required to shorten the time period required fromwhen an image is obtained until a strobe composite image is created,since the strobe composite image is often used in comments of, e.g., theform of a player in a television live program. Such requirements cannotbe met since it takes much time to create a strobe composite image usingthe conventional technique. Therefore, a method of generating a desiredstrobe composite image within a short period of time is demanded.

BRIEF SUMMARY OF THE INVENTION

[0009] The present invention has as its object to provide an imagecomposition method, apparatus, computer, and program product, which canreduce the load on the user upon generating a strobe composite image,and can shorten the time required for an image generation process.

[0010] A method, apparatus, computer, and program product according toan embodiment of the present invention generate a strobe composite imagefrom a plurality of frames of a moving image. A first frame is selectedfrom the plurality of frames of the moving image. A plurality of secondframes relating to the first frame are determined. Then, a strobecomposite image is generated by superposing the plurality of secondframes in accordance with a superposing manner.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

[0011]FIG. 1 is a schematic block diagram showing the arrangement of animage composition apparatus according to the first embodiment of thepresent invention;

[0012]FIG. 2 shows the superposing state of frames;

[0013]FIG. 3 shows a strobe composite image generated from a slow movingobject image;

[0014]FIG. 4 shows a strobe composite image generated by decimating someframes;

[0015]FIG. 5 is a flow chart showing a series of procedures of strobeimage composition according to the first embodiment;

[0016]FIG. 6 shows an example of an interface which comprises a sliderused to select a point of interest;

[0017]FIG. 7 shows an example of a window to be presented to the user todesignate a composition start point;

[0018]FIG. 8 is a flow chart showing the procedures for changing asuperposing manner by designating an objective range;

[0019]FIG. 9 shows an example of an interface which allows the user toselect two frames as start and end points;

[0020]FIG. 10 shows a slider operation state on the interface in FIG. 9;

[0021]FIG. 11 shows an end point setting state on the interface in FIG.9;

[0022]FIG. 12 shows an interface used to designate a range;

[0023]FIG. 13 is a view for explaining range designation by means of theinterface in FIG. 12, i.e., shows a state wherein the point of interestis moved to the start point;

[0024]FIG. 14 shows determination of the start point in rangedesignation;

[0025]FIG. 15 shows a state wherein the point of interest is moved tothe end point in range designation;

[0026]FIG. 16 shows determination of the end point in range designation;

[0027]FIG. 17 shows a state upon setting the second start point in rangedesignation;

[0028]FIG. 18 shows a state after the second start point is set in rangedesignation;

[0029]FIG. 19 shows selection of the second end point in rangedesignation;

[0030]FIG. 20 shows an example of storage data corresponding to thesetup contents of the superposing manner;

[0031]FIGS. 21A to 21C show modification examples of the superposingmanner;

[0032]FIG. 22 is a view for explaining a preview presentation interfaceaccording to the third embodiment of the present invention, i.e., showsa series of input video frames;

[0033]FIG. 23 shows one frame to be manually extracted;

[0034]FIG. 24 shows the manual extraction result of the frame shown inFIG. 23;

[0035]FIG. 25 shows a generation example of a strobe composite imageincluding the frame shown in FIG. 24;

[0036]FIG. 26 shows an example of a preview presentation interface;

[0037]FIG. 27 shows a state after a modification process on the previewpresentation interface;

[0038]FIG. 28 is a flow chart of a strobe image composition methodaccording to the fourth embodiment of the present invention;

[0039]FIG. 29 is a flow chart of a strobe image composition methodaccording to the fifth embodiment of the present invention;

[0040]FIG. 30 shows an example of a user interface used to make the userselect a desired strobe composite image;

[0041]FIG. 31 is a flow chart of a strobe image composition methodaccording to the sixth embodiment of the present invention;

[0042]FIG. 32 shows an example of a process for automatically extractinga region from an image for answer data;

[0043]FIG. 33 is a schematic block diagram showing the arrangement of astrobe image composition apparatus according to the seventh embodimentof the present invention;

[0044]FIG. 34 is a flow chart of a strobe image composition methodaccording to the eighth embodiment of the present invention;

[0045]FIG. 35 is a flow chart of a strobe image composition methodaccording to the ninth embodiment of the present invention;

[0046]FIG. 36 is a flow chart of a strobe image composition methodaccording to the 10th embodiment of the present invention;

[0047]FIG. 37 is a flow chart of a strobe image composition methodaccording to the 11th embodiment of the present invention;

[0048]FIGS. 38A and 38B show examples of feature point tracing results;

[0049]FIGS. 39A and 39B show the execution state of a motion patterndetermination step; and

[0050]FIG. 40 is a flow chart showing the flow of processes of a strobeimage composition method according to the 12th embodiment of the presentinvention.

DETAILED DESCRIPTION OF THE INVENTION

[0051] Embodiments of the present invention will be describedhereinafter with reference to the accompanying drawings.

[0052] (First Embodiment) FIG. 1 is a schematic block diagram showingthe arrangement of an image composition apparatus according to the firstembodiment of the present invention. As shown in FIG. 1, an imagecomposition apparatus of this embodiment comprises a display unit 1which displays an input video, various windows used to generate and editand a strobe composite image based on the input video, and the like, acentral processing unit (CPU) 2, an input unit 3 which comprises akeyboard, pointing device (mouse), and the like, and a main storage unit4. A program 5 used to generate and edit a strobe image, input videodata 6, and generated strobe image data 7 are recorded in an externalstorage device such as a hard disk device, magnetooptical disk device,or the like.

[0053] The image composition apparatus of this embodiment can beimplemented as software which runs on a versatile computer. In thiscase, the computer program 5 that implements the process for generatingand editing a strobe image is read out onto the main storage unit 4, andis executed by the CPU 2. Note that the arrangement of this embodimentincorporates an operating system (OS) that controls hardware of thecomputer and provides a file system and graphical user interface (GUI).In such arrangement, the computer program which implements the processfor generating and editing a strobe image is installed as applicationsoftware which runs on the operating system.

[0054] Prior to a detailed description of the processing operation ofthe image composition apparatus according to this embodiment, aschematic sequence of strobe composition will be explained. In strobecomposition, an input image is a moving image, and an output image is astrobe composite image (moving or still image). In order to generate anoutput strobe composite image from the input moving image, an objectextraction step of extracting only object portions which are to undergostrobe composition from the input image, and a step of designating astrobe composition object that represents frames and a manner ofsuperposing the frames so as to implement strobe composition arerequired. These two steps are independent from each other, and either ofthese steps can be executed first.

[0055] Let X be an input moving image of an object. Then, strobecomposite image Y is generated as follows. That is, the first frame ofmoving image Y is determined as an image of that of moving image X, andthe second frame of moving image Y is generated by superposing thesecond frame of moving image X on the first frame of moving image Y. Thethird frame of moving image Y is generated by superposing the thirdframe of moving image X on the second frame of moving image Y.Furthermore, the (k+1)-th (k is a natural number) frame of moving imageY is generated by superposing the (k+1)-th frame of moving image X onthe k-th frame of moving image Y. Then, appropriate times are assignedto respective frames of moving image Y to generate a moving image. FIG.2 shows respective frames of moving image Y.

[0056] In this case, frames are overlaid on each other. However, framescorresponding to a user's underlay instruction may be underlaid. Evenwhen only discrete frames are used, as will be described later, framesare overlaid or underlaid in accordance with the time series of inputobject images. The time series of input object images need not be thesame as that upon photographing an object. For example, the reverseorder of the time series, i.e., reverse playback, may be used.

[0057] The frames to be superposed need not always be consecutive, butmay be discrete. Especially, the user often wants to select every N-thframes (N is an integer) to be superposed. Hence, a user interface thatallows the user to select every N-th frames to be superposed from theA-th to B-th frames (A, B, and N are integers) may be provided.

[0058] For example, when every other frames are selected as frames to besuperposed like frame Nos. 1, 3, 5, 7, and 9 from a moving imageincluding 10 frames from frame Nos. 0 to 9, a composite image can begenerated from only odd frames of an interlaced-scan moving image.

[0059] As another example, when an object moves relatively slowly, ifall frames are selected as those to be superposed, a strobe compositeimage is complicated (e.g., FIG. 3). In such case, when some frames aredecimated (e.g., every other frames are selected as those to besuperposed), a plain strobe composite image is obtained, as shown inFIG. 4.

[0060] An interface which makes the user designate one frame and alsothe superposing manner to composite a strobe image is provided. Usingthis interface, a series of procedures shown in FIG. 5 are executed.Initially, a frame select/input procedure is executed (step S1) to waitfor one frame selected by the user. If the user has selected one frame,a frame shift procedure for calculating and determining an objectiveframe which is commonly called as an IN or OUT point is executed (stepS2). The objective frame obtained by the frame shift procedure undergoesa superposing manner setting procedure (step S3). Finally, a superposingprocedure is executed to obtain a strobe composite image (step S4). Inthe frame select/input procedure (step S1), as shown in, e.g., FIG. 6,an interface that displays a slider 3201 and the currently selectedframe 3202 is presented to the user. On this interface, the user canselect a desired frame by operating the slider 3201 using a mouse orkeyboard.

[0061] In the frame select/input procedure, a plurality of frames may bepresented. For example, when the currently selected frame (frame 1402 ofinterest) and frames 1401 and 1403 before and after that frame aredisplayed, as shown in FIG. 7, if the interface is designed to allow theuser to select a frame not only by means of the slider but also byclicking a frame itself (1401 to 1403), the frame selection operationcan become more intuitive and easier to use for the user. Alternatively,frames obtained as a result of the frame shift procedure of thecurrently selected frame are preferably presented. As a result, the usercan immediately recognize frames designated by the currently selectedcontents.

[0062] In the superposing manner setting procedure (step S3), forexample, the objective frame is set as one of the start and end pointsof strobe composition. When strobe composition is made by superposing Nconsecutive frames (N is a natural number) from a specific start frameto a specific end frame on (or under) the next frames in turn, the startframe is set as the start point of strobe composition, and the end frameis set as the end point. Note that frames need not always beconsecutive, and the start and end points may be similarly determinedfor an arbitrary number of discrete frames. Also, based on user's frameselection, a plurality of frame shift procedures and superposing mannersetting procedures may be simultaneously executed. For example, both thestart and end points of strobe composition may be simultaneouslydetermined based on one designated frame.

[0063] A method of designating the start point of strobe compositionwill be described below with reference to FIG. 7. FIG. 7 shows anexample of a window to be presented to the user to designate the startpoint. In this embodiment, assume that M frames from a frame N framesbefore the frame of interest are to undergo strobe composition, and Nand M (integers) are determined in advance. When the user has selectedthe frame 1402 of interest (frame No. 71 in this example), it isdetermined that overlay strobe composition is designated to have a frameN frames before the frame of interest (frame No. 65 in this example) asa start frame (start point), and a frame M frames after the start frame(frame No. 78 in this example) as an end frame (end point). When theuser presses a “start point determination” button, composition isexecuted.

[0064] Note that execution of strobe composition need not always bestarted in response to depression of the “start point determination”button. For example, an interface may be designed to determine that an“overlay” button is pressed while the start to end frames have beenselected, and to wait for a composition execution instruction. An imageto be presented may be either an extracted object or an image beforeextraction. In the above example, the end frame is defined. However,only the start point may be selected without determining M.

[0065] (Second Embodiment) An interface which makes the user designatetwo frames and also the superposing manner to implement strobe imagecomposition will be explained below. Based on this interface, theprocedures shown in FIG. 8 are executed. In order to quickly obtain astrobe composite image by minimum required user operations, the firstembodiment is preferably used. However, when the user wants to strictlydesignate the start and end points (frames) of strobe composition, theinterface of the second embodiment is effective. In this embodiment, thesuperposing manner of some frames of the already created strobecomposite image can be changed to generate an image again.

[0066] The frame select/input procedure is executed to wait for twoframes (start and end points) selected by the user in this embodiment(step S11). If the user has selected the frames, frames between the twoselected frames are determined as those to be processed, and aresuperposed (step S12). In this embodiment, the superposing manner ischanged (step S13), and the superposing procedure is executed again toobtain a strobe composite image (step S14).

[0067] An interface which presents a frame select window presents oneslider 3301, and a frame 3302 to be selected by this slider 3301, asshown in, e.g., FIG. 9, and makes the user select two frames from theslider. Buttons 3303 and 3304 are used to set the current point ofinterest as the start and end points, respectively. FIG. 9 illustrates astate wherein the user has pressed the button 3303 to set the point ofinterest (frame No. 45) as the start point. The user moves the slider3301, as shown in FIG. 10, and sets the point of interest (frame No. 71)as the end point using the button 3304, as shown in FIG. 11. As in thefirst embodiment, when the user selects a frame using the slider 3301,not only the currently selected frame but also frames before and afterthat frame may be presented, and the user may select a frame by directlyclicking it in addition to selection using the slider 3301. Thus, theframe selection operation becomes more intuitive and easier to use.

[0068] As the superposing manner in step S12, an arbitrary method may beused. For example, frames may be superposed in turn in ascending ordescending order of time stamp.

[0069] A case will be exemplified below wherein a designated range is tobe superposed in association with the series of procedures shown in theflow chart of FIG. 8. FIG. 12 shows an example of a window to bepresented to the user in this case. The interface to be described belowmakes the user select the first frame of objective frames using a button3004 with text, and also select the last frame of the objective framesusing a button 3005 with text. In FIG. 12, reference numerals 3001 and3003 denote fields which display frame images in correspondence with thebuttons 3004 and 3005. With these images, the user can recognize thecontents of the currently selected frames at a glance. A button 3006with text displays the frame number of the current frame of interest,and a field 3002 displays an image corresponding to that frame number.The user can move the point of interest by moving a slider 3013 to theright or left by means of a mouse or keyboard input. Prior to selectionof frames, the user can designate a superposing manner using exclusivepush buttons 3010 and 3011. These buttons are mutually exclusive: whenthe user presses one of these buttons, the other button is restored to anon-pressed state.

[0070] FIGS. 13 to 19 show the buttons 3004, 3005, and 3006, and theslider 3013 extracted from FIG. 12. A series of operations for selectingthe first and last frames to be selected will be described below withreference to these drawings. Display objects of the buttons 3004, 3005,and 3006 can display text that indicates button names, and numericalvalues that indicate the frame positions (numbers) (these buttons willbe referred to as “buttons with text” hereinafter). Note that thedisplay contents of a numerical value on each button are updated inaccordance with an operation of the slider 3013 and a settlementinstruction for that button.

[0071] When the interface is presented to the user for the first time,neither a start point nor end point are designated at that time. In thisstate, the end point is grayed out, and its button cannot be pressed, asshown in FIG. 13. The user moves the point of interest to a frame thathe or she wants to set as the start point. The user then presses thebutton 3004 with text of the start point to set the current point ofinterest as the start point, as shown in FIG. 14.

[0072] The user moves the point of interest to a frame that he or shewants to set as the end point, as shown in FIG. 15. At this time, sinceobjective frames are not settled, they are indicated in light color.When the user presses the button 3005 with text of the end point, theend point is settled. The user can make superposing designation of theobjective frames by pressing one of the exclusive push buttons 3010 and3011, which are used to designate the superposing manner at that time.When the user presses the button 3005 with text of the end point onceagain in the state shown in FIG. 16, the settled contents are canceledto revert the state shown in FIG. 15. When the user presses a“composition execution” button 3014 in the state shown in FIG. 16,strobe composition is executed. On the other hand, when the user selectsthe frame of interest again in the state shown in FIG. 16, and pressesthe start point button again, as shown in FIG. 17, he or she can makesuperposing designation of another strobe image for a new period aftersuperposing designation made so far, as shown in FIG. 18. The user makesthe same objective frame selection operations as those described above,i.e., selects the end point of this new period, as shown in FIG. 19,changes the depression states of the exclusive push buttons 3010 and3011 as needed, and so forth, thus making superposing designation ofanother set of objective frames.

[0073] A button 3007 is used to store the current superposing setupcontents in the storage unit 4. For example, the superposing setupcontents indicate superposing processes to be applied to frames atrelative positions from the current point of interest, as shown in FIG.20. When the user selects the point of interest and presses a button3012, the saved superposing setup contents are loaded and applied. Bycalling the saved setup in this way, the edit time can be shortened.Some pre-stored different setups may be prepared as a list, and thatlist may be presented in response to a user's request to load anothersetup. In this way, a plurality of different setups can be preferablyselectively used.

[0074] The current superposing setup contents stored in the storage unit4 may be permanently recorded using a recording unit such as an HDD orthe like, and may be read out and applied to composition of anotherstrobe image. That is, the readout superposing setup may be applied to aframe group, different from the currently processed frame group later,to composite another strobe image.

[0075] Also, a plurality of superposing setups with different contentsare recorded in the recording unit to simultaneously generate strobecomposite images corresponding to these superposing setups. Furthermore,a plurality of strobe composite images may be generated on the basis ofone arbitrary superposing setup. It is preferable to make onlysuperposing setup processes in advance, and to simultaneously executeimage generation processes later in terms of high work efficiency.

[0076] An interface which is used when a strobe composite image isgenerated and presented once by overlaying all frames, and is generatedagain using objects designated by the user from these frames whilechanging the superposing manner to the underlay manner will be describedbelow. Such processes correspond to the procedures in steps S13 and S14in FIG. 8.

[0077] As shown in FIGS. 21A to 21C, a composite image is displayed byoverlaying frames (FIG. 21A). In this state, the user selects, e.g.,objects 2101 of three left object images by clicking the left button ofa mouse. The selected objects 2101 are preferably displayed in a colordifferent from other objects (FIG. 21B). When the user clicks the rightbutton of the mouse, the superposing manner of these three selectedobjects 2101 is switched to the underlay manner, and a new compositeimage is displayed, as shown in FIG. 21C. In place of presenting allframes as an overlaid image, the interface shown in FIG. 12 may bepresented in advance. Then, when the user clicks the right button of themouse, overlaid objects may be changed to underlaid objects, and viceversa. In this way, two processes, i.e., designation of a roughsuperposing manner, and adjustment of superposing manners for respectiveframes can be attained within a short period of time.

[0078] In this embodiment, the interface which allows the user to selectobjective frames from a strobe composite image, and to additionallydesignate to change their superposing manner has been explained. If aninterface which can also change the superposing order when the userselects objective frames from a strobe composite image is added, thetime required to create and edit a strobe composite image can be furthershortened. Note that changing a strobe composite image at the instanceof clicking requires high processing power of the image compositionapparatus, and also operator's skills to cope with an instantaneouschange in screen. Hence, such process is preferably selectively usedaccording to situations.

[0079] In the above description, a frame is selected using the slider.Alternatively, for example, some representative frames may be presentedto prompt the user to select one of these representative frames. Then,only frames near the selected representative frame may be presented toprompt the user to input a superposing instruction. In this way, theaforementioned slider operation can be omitted.

[0080] As described above, since the interface which can minimize user'soperations required for strobe composition is provided, the strobecomposition execution sequence can be clarified, and the operation loadon the user can be greatly reduced. Therefore, the time required tocreate a strobe composite image can be shortened, and the usability ofthe image composition system can be improved.

[0081] (Third Embodiment) The third embodiment which comprises aninterface that presents a preview during a manual extraction processwill be described below with reference to FIGS. 22 to 27. A situation inthe middle of a user's manual extraction process of an object image frominput video frames Ia to Ic shown in FIG. 22 will be examined. Note thatthe object image is a fish that is swimming from the right to the lefton the plane of the drawing. The user has already extracted objectimages from the first and second frames Ia and Ib. The user is about toextract an object image from the third frame Ic.

[0082] As shown in FIG. 23, the user is extracting an object image 1702so as not to include an object image 1701. If a portion 1801 of theimage is inaccurately extracted, as shown in FIG. 24, its influenceappears as an unwanted portion 1901 in a strobe composite image, asshown in FIG. 25, resulting in deterioration of image quality of thestrobe composite image.

[0083] As shown in FIG. 26, in addition to the input image and extractedimage, an image 1803 after strobe composition based on a currentlyextracted image 1802 is sequentially superposed and is displayed as apreview. The user can immediately confirm a portion which is extractedinaccurately and adversely influences a strobe composite image, and caneasily correct such portion to obtain a desired strobe composite image.As shown in FIG. 27, the user can immediately confirm a corrected strobecomposite image 1804.

[0084] Note that “preview” amounts to presenting the processing resultbased on the current instruction contents before actual processing, andsequentially updating and presenting a complete image (may be either astill or moving image) in accordance with, e.g., frames to be superposedselected by the user.

[0085] Note that not all of the input image, extracted image, and strobecomposite image need always be presented, and only two images, e.g., theinput image and strobe composite image may be presented. Also, a previewmay be presented not only when the user manually extracts an objectimage but also when the user corrects an automatic or semi-automaticextraction result of software.

[0086] When an object is extracted from an image region such as abackground or the like, which does not change along with the elapse oftime, inaccurate extraction does not influence a strobe composite image,and the user need not spend much time for such extraction. When apreview is presented, the user can confirm it. Therefore, the timerequired to generate a strobe composite image can be shorted whilesaving labor.

[0087] The user can recognize a final image during an edit process, andneed not check if a strobe composite image is obtained as he or she hadexpected after generation of it. In this way, sequential presentation ofa strobe composite image as a preview greatly contributes to shorteningthe time required to generate the strobe composite image and reductionof the load on the user compared to presentation of an extracted imageas a preview.

[0088] (Fourth Embodiment) FIG. 28 is a flow chart showing a series ofprocesses of strobe image composition according to the fourth embodimentof the present invention. In this embodiment, upon reception of a user'sinstruction (example: depression of a button, clicking of a mousebutton, and the like) during capturing a moving image, generation of astrobe composite image starts in response to that instruction as atrigger. An input device is, for example, a camera which captures amoving image, and is connected to a strobe composition system when it isused. A system that practices the fourth embodiment will be referred toas a 1-instruction type strobe composition system hereinafter. Thissystem has a queue (first-in, first-out (FIFO) buffer) for saving amoving image. Note that a multi-stage delay circuit may be used in placeof the queue.

[0089] The 1-instruction type strobe composition system repeats stepsS3401 and S3402 shown in FIG. 28 unless an one-click-instruction isdetected. This instruction may be done by clicking a mouse or switch bythe user, and is referred to as “one-click-instruction”. An image inputstep (S3401) of capturing the next image frame from the input device,and an image holding step (S3402) of shifting the queued frames by oneframe by discarding the first frame in the queue, and adding thecaptured frame as the last frame are repeated.

[0090] Upon detection of an one-click-instruction (S3403: user inputstep), the 1-instruction type strobe composition system sets a timerwhich generates notification (time out) at least once when apredetermined period of time elapses after the detection time of theinstruction (S3403), and proceeds with processes in steps S3401 andS3402. If the timer generates notification (S3405), the system executesstrobe composition using images saved in the queue (S3406): imagecomposition step) and outputs a composite image (S3407: image outputstep). According to this embodiment, since the user need only input oneinstruction alone, the load on the user can be greatly reduced, and thetime required to generate a strobe composite image can also be greatlyshortened.

[0091] (Fifth Embodiment) FIG. 29 is a flow chart showing a series ofprocesses of strobe image composition according to the fifth embodiment.This embodiment automatically detects a frame which is important uponexecuting strobe composition (to be referred to as a “feature imageframe” hereinafter) during capturing of a moving image, and generationof a strobe composite image starts in response to that detection as atrigger. Such system of the fifth embodiment will be referred to as a“fully automatic strobe composition system” hereinafter.

[0092] As in the fourth embodiment, the fully automatic strobecomposition system according to this embodiment has a queue (FIFObuffer) for saving a moving image. This system repeats steps S3501 toS3503 to be described below. That is, the system repeats an image inputstep (S3501) of capturing the next image frame from the input device, animage holding step (S3502) of shifting the queued frames by one frame bydiscarding the first frame in the queue, and adding the captured frameas the last frame, and a feature image frame detection step (S3503) ofchecking whether or not a feature image frame is present.

[0093] If a feature image frame is present, the fully automatic strobecomposition system sets a timer which generates notification (time out)at least once when a predetermined period of time elapses after thedetection time of the feature frame (S3504), and proceeds with processesin steps S3501 to S3503. If the timer generates notification (S3505),the system executes strobe composition using images saved in the queue(S3506): image composition step) and outputs a composite image (S3507:image output step).

[0094] As described above, according to this embodiment, no user'sinstruction is required, and the time required to generate a strobecomposite image can be further shortened compared to the fourthembodiment. In the fourth and fifth embodiments that detect a user'sinstruction or feature image frame, the timer is used. Alternatively,times may be recorded at given time intervals, and an elapse of apredetermined period of time may be detected. In the fully automaticstrobe composition system, various methods of determining a featureimage frame and detecting such frame may be used. Such methods will beexplained taking a strobe composite upon hitting a tee shot at golf asan example. For example, strobe image composition conformity conditionsinclude:

[0095] (1) A sound produced upon hitting a tee shot is detected, and animage frame at the time of detection of the sound is determined as afeature image frame.

[0096] (2) An object shape (template) in a feature image frame isprepared in advance, an object region is extracted from each imageframe, and when it is determined that the extracted object shape issufficiently similar to the template, that frame is determined as afeature image frame.

[0097] (3) Templates of an object for a feature image frame and severalframes before and after the feature image frame are prepared in advance,an object region is extracted from each image frame, and when it isdetermined that the shapes of the object for several successive framesare sufficiently similar to the templates, that frame is determined as afeature image frame.

[0098] (4) The area of an object region for each image frame iscalculated, the difference (peak-to-peak value) between the maximum andminimum areas of several successive frames is calculated, and when thepeak-to-peak value has exceeded a value, which is set separately, aframe that gives the maximum area (or minimum area) is determined as afeature image frame.

[0099] Upon extracting an object region, a monochrome background may beprepared in advance, and the chroma-key method that extracts a colorregion different from the background color as an object region may beused. Upon calculating the area of an object region, the object regionmay be extracted using, e.g., the chroma-key method, and the number ofpixels of that region may be calculated as the area. As criteria usedupon determining a similarity with a template image, for example, anarea St of an object region of a template, an area Se of an extractedobject region, and an area Sc of overlapping region between the objectregion of the template and the extracted object region are calculated,and when a smaller one of two values (Sc/St) and (Sc/Se) is equal to orlarger than a threshold value, a high similarity may be determined.

[0100] A method suited to an object of target strobe composition isselected in advance from these methods, and a feature image frame isdetected by the selected method, thus improving the detection precisionof a feature image frame. Therefore, the serviceability of the fullyautomatic strobe composition system can be improved. As a result, thetime required to generate a strobe composite image can be shortened.

[0101] (Present Preset)

[0102] The quality of a strobe composite image to be generated by the1-instruction type strobe composition system or fully automatic strobecomposition system depends on the types of input images and the strobecomposition algorithm. For example, when an object region of eachobjective frame is extracted by the chroma-key method, a threshold valuewhich sets a background color range in a color space must be controlled.If the user can set a parameter such as a threshold value in thechroma-key method or the like, high quality can be obtained. However,when the user must set such parameter for each strobe composition, theload on user's operations increases, and the time required for strobecomposition is prolonged. Hence, some different parameters are preparedin advance, and the user can preferably select a parameter to be usedbefore strobe composition. In this manner, the quality of a strobecomposite image can be improved, and high work efficiency can beassured.

[0103] (Present Last Frame of Strobe Image or Playback Moving Image)

[0104] It is often difficult for some input images to determine a strobecomposition parameter in advance. In this case, parameter candidates arelimited to several types, and strobe composition is made using all thesecandidates. Then, the obtained results are presented to prompt the userto select one of these results. In this way, the user can obtain adesired output image. For example, on a user interface using a mouse anddisplay screen, strobe composition is made using three differentparameters, and the user selects a desired strobe composite image fromlast frames of strobe composition presented on fields 3601, 3602, and3603 by mouse operation, as shown in FIG. 30. In place of the lastframes, strobe composite moving images may be played back. With thismethod, even when sufficiently high quality cannot be achieved by user'sparameter setup operation, the user's operation can be minimized, andthe time required for strobe composition can be consequently shortened.

[0105] (Sixth Embodiment) FIG. 31 is a flow chart of strobe imagecomposition according to the sixth embodiment of the present invention.In this embodiment, automatic extraction parameters used in strobe imagecomposition are determined in advance. This embodiment will take a sportgame as an example, and will explain a case wherein a strobe image isgenerated based on capturing an image of a player during that game.

[0106] Before the game, a moving image of a player (object) is capturedinto a personal computer as answer data (S4101). That moving image isdisplayed on the display of the personal computer while being frozenframe by frame, and the user accurately inputs a player region of eachframe using a mouse or the like (S4102). Images obtained by extractingthe player regions are superposed while being overlaid frame by frame,thus generating a strobe composite image (S4103). This strobe compositeimage has very high composition quality, and will be referred to as ananswer strobe composite image hereinafter.

[0107] A player region is automatically extracted from the answer dataimage. FIG. 32 shows an example of the extraction method. The differencebetween the luminance values of a frame 4301 to be extracted and anotherreference frame 4302 is calculated. If this inter-frame difference islarger than a threshold value, an object region is determined;otherwise, a background region is determined, thus generating an alphamask 4303. The alpha mask is a bitmap comprising two values, i.e., theobject region and background region.

[0108] A plurality of alpha masks 4305 based on differences from otherreference frames 4303 are generated, and their object regions arelocally ANDed to obtain an alpha mask 4306 of an accurate object region.Finally, the obtained alpha mask undergoes profile fitting using amethod described in reference “Profile Fitting based on Self-similarMapping” (Ida, et al., 5th Image Sensing Symposium lecture papers C-15,pp. 115-120, June 1999) or the like, thus obtaining a more accuratealpha mask 4307. Profile fitting is a profile extraction process forobtaining a profile which suffers less errors from a profile whichsuffers many errors.

[0109] In case of the aforementioned extraction method according to thisembodiment, extraction parameters include a threshold value used todetermine an object or background region, a frame interval between theobjective frame and reference frame, profile fitting strength, and thelike.

[0110] The extraction parameters are temporarily set to certain values,and player regions are extracted (S4104). Based on the extractionresults of the player regions, a temporary strobe composite image isgenerated (S4105). The temporary strobe composite image normally haspoorer quality than the answer strobe composite image. In order toestimate the degree of quality deterioration, errors of the temporarystrobe composite image from the answer strobe composite image arecalculated (S4106). Some combinations of parameters may be determined,and steps S4104 to S4106 are repeated while switching parameters withinthe range of these combinations. That is, if all predeterminedparameters are used, the flow advances to the next step; otherwise, theflow returns to step S4104 (S4107).

[0111] After steps S4104 to S4106 are repeated for all the parameters,parameters which correspond to minimum errors of the temporary strobecomposite image from the answer strobe composite image are formallydetermined as those which are applied to the subsequent process (S4108).The parameters determined in this step will be referred to as executionparameters in this embodiment. Object regions are extracted from imagesother than the answer data images, which are captured during a game(S4109), and a strobe composite image is generated (S4110).

[0112] Optimal values of the extraction parameters depend onenvironmental changes such as a player's uniform color, backgroundcolor, noise feature produced upon capturing an image by a camera, andthe like. However, these environments remain the same during anidentical game, and satisfactory composition can be attained withoutswitching parameter values. According to this embodiment, parametervalues which can obtain a satisfactory composite image even during agame can be determined. In this manner, automatic composition that doesnot require any manual operations can be implemented, and a strobecomposite image can be generated within a short period of time.

[0113] Note that a characteristic feature of this embodiment is toevaluate errors using a composite image in place of an extracted regionshape. As described in the third embodiment, in case of strobe imagecomposition, region shape errors do not influence a composite image.Even when a background portion is temporarily extracted as a playerregion, an identical background image is overwritten on the backgroundportion which remains the same, thus obtaining the same compositionresult as that obtained when this portion is not detected as a playerregion. As the errors, for example, differences may be calculatedbetween corresponding frames, and the sum or square sum of the absolutevalues of the differences may be used.

[0114] (Seventh Embodiment) FIG. 33 is a schematic block diagram showingthe arrangement of a strobe image composition apparatus according to theseventh embodiment of the present invention. The seventh embodimentrelates to a practical apparatus which executes a series of processes ofstrobe image composition that have been explained in the sixthembodiment.

[0115] A captured image 4201 is input, and is held in a storage unit4202 such as a semiconductor storage element, magnetic recording device,or the like in case of, e.g., a personal computer. An image 4203 is sentas an answer data image to a setting unit 4202 which sets an answerobject region. For example, the image 4203 is displayed on the displayof the personal computer, and the user accurately inputs an objectregion using a mouse or the like. An answer object image 4205 extractedbased on the input region is sent to a composition unit 4206. Thecomposition unit 4206 overwrites the object image 4205 in the order oftime, and sends an answer strobe composite image 4207 to a holding unit4208.

[0116] On the other hand, the image 4203 is also sent to an objectregion detection unit 4212. The detection unit 4212 detects an objectregion on the basis of extraction parameter values 4211 sent from aparameter setting unit 4210, and an object image 4213 extracted based onthat region is sent to a composition unit 4214. The composition unit4212 overwrites the object image 4213 in the order of time, and sends astrobe composite image 4215 to an error detection unit 4216. The answerstrobe composite image 4207 is sent from the holding unit 4208 to theerror detection unit 4216. The error detection unit 4216 calculateserrors 4217 between the answer strobe composite image 4207 and strobecomposite image 4215. The calculated errors 4217 are sent to theparameter setting unit 4210.

[0117] The parameter setting unit 4210 sequentially switches theparameter values 4211, checks the result obtained based on the switchedparameter values, and obtains parameter values which can minimize theerrors 4217. When another image 4201 is input, the parameter values 4211are sent to the detection unit 4212, which detects an object region froman image 4203 read out from the storage unit 4202. An object image 4213undergoes strobe image composition by the composition unit 4214. Astrobe composite image 4215 obtained as a result of this process isexternally output.

[0118] With this arrangement, the second and subsequent input images canbe automatically processed, and a strobe composite image can begenerated within a short period of time.

[0119] (Eighth Embodiment) The eighth embodiment of the presentinvention allows the user to select an image frame used in compositionof a strobe image while observing a slow playback video of an objectmoving image. The eighth embodiment is executed from step S3701 inaccordance with a series of procedures shown in FIG. 34. Prior to adescription of contents executed in respective steps, a term“superposing manner switching frame” is used as follows. Morespecifically, upon generating a strobe composite image by superposing aplurality of frames of a moving image, an image frame at which asuperposing manner (overlay or underlay composition) is switched will bereferred to as a “superposing manner switching frame” hereinafter. Thecontents executed in respective steps are as follows.

[0120] Step S3701: One image frame is captured and displayed. The flowthen advances to step S3702.

[0121] Step S3702: A user's image frame selection instruction isaccepted. If no instruction is detected within a predetermined period oftime, the flow returns to step S3701. Upon detection of an instruction,the flow advances to step S3703.

[0122] Step S3703: The image frame displayed in step S3701 immediatelybefore the user's instruction accepted in step S3702 is determined as asuperposing manner switching frame. A position shifted from the positionof that superposing manner switching frame by a predetermined value inthe time direction is determined as a start point. Also, a positionshifted by another predetermined value is determined as an end point.Then, a predetermined number of image frames at predetermined intervalsof those from the start point to the end point are determined asobjective image frames which are to undergo strobe composition. Afterthat, the flow advances to step S3704.

[0123] Step S3704: If image frames to be captured of those which aredetermined in step S3703 and are to undergo strobe composition stillremain, these frames are captured. The flow then advances to step S3705.

[0124] Step S3705: The image frames which are determined in step S3703and are to undergo strobe composition are superposed to generate astrobe composite image. A series of procedures end.

[0125] Conventionally, the user cannot often accurately select a desiredimage frame while observing a video played back at a frame rate uniqueto a moving image of an object if he or she makes such attempts.However, according to this embodiment, the user can accurately select adesired image frame. As a result, the time required until a strobecomposite image can be shortened.

[0126] Note that this embodiment can be modified as follows.

[0127] (1) When the user issues a wrong instruction to be accepted instep S3702, he or she must quit the series of procedures and redo themto correct errors. In order to allow the user to correct such errors, itis preferable that an image displayed when the user has found errorsrevert to a previously displayed image, and wrong operations be redoneretrospectively.

[0128] (2) In step S3701, an image is displayed. If the user candetermine the image display time, user errors due to an instructionaccepted in step S3702 can be reduced. The display time may be changedduring image display.

[0129] (3) One image frame is sequentially captured in step S3701, and arequired number of image frames are also captured in step S3704. Inplace of these processes, image frames may be simultaneously captured,or a plurality of image frames may be captured in step S3701.

[0130] (4) The superposing manner switching frame, and the start and endpoints are determined in step S3703 in accordance with the user'sinstruction accepted in step S3702. In this case, an image framecorresponding to the user's instruction is determined as a superposingmanner switching frame. Alternatively, for example, an image framecorresponding to the user's instruction may be determined as the startor end point.

[0131] (5) When the overlay and underlay composition modes need not beswitched, the superposing manner switching frame need not be determined.Hence, in such case, the start and end points may be automaticallydetermined with reference to an image frame corresponding to the user'sinstruction. In this way, the number of times of user's operations canbe reduced, and the time required until a strobe composite image can beshortened.

[0132] (6) The superposing manner switching frame, and the start and endpoints are determined in accordance with one user's instruction acceptedin step S3702. In order to obtain a strobe composite image that the userwants, one image frame may be determined based on one user's instructionin place of a plurality of image frames. Such modification is effectivewhen a strobe composite image with sufficiently high quality cannot begenerated based on only one user's instruction, and this embodiment canbe applied to more images.

[0133] (7) When the start point alone is determined but the end point isnot determined yet, and when execution of step S3705 starts immediatelyafter the start point is determined, the time required fromdetermination of the end point until completion of strobe compositioncan be shortened.

[0134] (Ninth Embodiment) The ninth embodiment of the present inventionallows the user to select an image frame while observing a playbackvideo of an object strobe composite moving image. The ninth embodimentis executed according to a series of procedures shown in FIG. 35. Theseries of procedures are executed from step S3801. The contents executedin respective steps are as follows.

[0135] Step S3801: One image frame is captured and displayed. The flowthen advances to step S3802.

[0136] Step S3802. A user's strobe composition start instruction isaccepted. If no instruction is detected within a predetermined period oftime, the flow returns to step S3801. Upon detection of an instruction,the flow advances to step S3803.

[0137] Step S3803: One frame image is captured. The flow then advancesto step S3804.

[0138] Step S3804: A user's instruction which indicates if a superposingmanner (overlay or underlay composition) is to be changed upongenerating a strobe composite image by superposing image frames isaccepted. If a change instruction is detected, the flow advances to stepS3805. If no instruction is detected within a predetermined period oftime, the flow jumps to step S3806. Or if an instruction indicating thatthe method is not changed is detected, the flow jumps to step S3806.

[0139] Step S3805: A superposing manner (overlay or underlaycomposition) used upon generating a strobe composite image bysuperposing image frames is changed. After that, the flow advances tostep S3806.

[0140] Step S3806: The image frames are superposed in accordance withthe superposing manner to generate a strobe composite image. After that,the flow advances to step S3807.

[0141] Step S3807: The strobe composite image generated in step S3806 isdisplayed. The flow then advances to step S3808.

[0142] Step S3808: A user's instruction indicating if strobe compositionis to quit is accepted. If no instruction is detected within apredetermined period of time, the flow returns to step S3803. Or if aninstruction indicating that the process does not quit is detected, theflow returns to step S3803. If a quit instruction is detected, theseries of procedures quit.

[0143] According to this embodiment, since strobe composition, playbackof a strobe composite moving image, and selection of image frames can besimultaneously done, the time required until strobe composition that theuser wants is completed can be shortened.

[0144] Note that the ninth embodiment can be modified as follows.

[0145] (1) When the user issues a wrong instruction to be accepted instep S3802, S804, or S3808, he or she must quit the series of proceduresand redo them to correct errors. In order to allow the user to correctsuch errors, it is preferable that an image displayed when the user hasfound errors revert to a previously displayed image, and wrongoperations be redone retrospectively.

[0146] (2) In steps S3801 and S3807, an image is displayed. If the usercan determine the display time of these images, user errors due toinstructions accepted in steps S3802, S3804, and S3808 can be reduced.The display time may be changed during image display.

[0147] (3) One image frame is sequentially captured in each of stepsS3801 and S3803. For example, image frames may be simultaneouslycaptured, or a plurality of image frames may be captured in steps S3801and S3803.

[0148] (4) Upon accepting a user's instruction in step S3804, thesuperposing manner upon strobe composition is changed. A positionshifted from the position of an image frame corresponding to that user'sinstruction in the time direction may be determined as an end point. Inthis case, when an image frame corresponding to that end point issuperposed to generate a strobe composite image, and that strobecomposite image is displayed, the series of procedures end. In this way,one of user's instructions can be omitted, and the time required untilstrobe composition is completed can be shortened.

[0149] (5) The position shifted in the time direction may be determinedas a start point in place of the end point. In this way, if image framesat positions shifted from the position of an image frame correspondingto a given user's input are determined as those corresponding to thestart and end points, and a frame as the boundary upon switching thesuperposing manner, even when the user's instruction accepted in stepS3802 includes wrong contents, operations need not be redone orcorrected, thus shortening the time required for strobe composition.

[0150] (6) In the ninth embodiment, a strobe composite image isdisplayed simultaneously with strobe composition. The strobe compositeimage may be stored in a storage unit, and may be played back later.

[0151] (10th Embodiment) FIG. 36 is a flow chart showing the flow ofprocesses of a strobe image composition method according to the 10thembodiment of the present invention. Referring to step S36, S4601 is animage input step of sequentially inputting time-serial image frames;S4602, an image holding step of holding latest N image frames (N is anatural number) input in the image input step; S4603, a reference imageframe detection step of detecting a reference image frame serving as areference for a plurality of image frames, which are to undergo strobeimage composition, on the basis of strobe image composition conformityconditions used to determine if a given image frame conforms to strobeimage composition; S4604, an image composition step of executing astrobe image composition process for the plurality of image framesstored in the image holding step when a predetermined period of time iselapsed after detection of the reference image frame; and S4605, animage output step of outputting a composite image frame that hasundergone the strobe image composition process in the image compositionstep.

[0152] The image input step S4601 corresponds to a step of capturing andinputting an object image which is to undergo strobe composition likecamera shooting in a sports live program. In this step, time-serialimage frames are input sequentially. The image holding step S4602 isholding the latest N image frames input in the image input step S4601.For example, frames for latest 2 sec (60 frames at 30 [fps]) of a sportslive video are held while always updating the held contents of a storagemedium with a limited capacity.

[0153] The reference image frame detection step S4603 detects areference image frame serving as a reference for a plurality of imageframes, which are to undergo strobe image composition, on the basis ofthe strobe image composition conformity conditions used to determine ifthe currently held image sequence are image frames suited to strobeimage composition while the latest N images are held in the imageholding step S4602. For example, when video frames of a baseball liveprogram are input in the image input step S4601, the image sequencewhich is to undergo the strobe composition process includes a pitchingscene of a pitcher, a hitting scene of a batter, and a plurality ofimage frames of a ball. Note that the reference image frame correspondsto a frame at the instance when a pitcher releases a ball or a frame atthe instance when a batter hits a ball, and a desired strobe compositeimage can be generated by superposing frames before and after thatreference image frame. The strobe image composition conformityconditions include, e.g., a condition using image information, and thatusing additional information different from image information. Thecondition using image information uses information only in images. Onthe other hand, the condition using additional information differentfrom image information obtains additional information different fromimage information in another input step synchronous with the image inputstep. In this case, a microphone, infrared sensor, scale, or physicalswitch is used. Based on these two conditions, a reference image framecan be detected as follows.

[0154] (Condition Using Image Information)

[0155] A frame from which an image frame similar to an image, which isregistered in advance, is detected is detected as a reference imageframe. For example, the image registered in advance is an image frame atthe instance of impact.

[0156] A frame from which a change in pixel value of a partial region,which is set in a frame in advance, is detected is detected as areference image frame. For example, when a position where a golf ball isplaced is set as the partial region, a frame in which white pixels havechanged to green pixels is detected as the reference image frame. Also,a frame when a baseball bat has passed a given region is detected as thereference image frame.

[0157] When a mask image similar to a mask image which is set in advance(a binary image used to separately extract an object region andbackground region) is generated, an original image frame of this maskimage is detected as a reference image frame.

[0158] A frame which has a minimum area of an object region in a maskimage generated from an image frame is detected as a reference imageframe. A hitting scene of baseball or a swing scene of golf normally hasa minimum area of an object region at the instance of impact.

[0159] (Condition Using Additional Information Different from ImageInformation)

[0160] Upon detection of a sound which is set in advance, a framecorresponding to that detection timing is detected as a reference imageframe. The sound to be detected includes a hitting sound of baseball, animpact sound of golf, and the like.

[0161] When the infrared sensor detects an object to be detected, aframe corresponding that detection timing is detected as a referenceimage frame. For example, an image frame corresponding to a timing atwhich the infrared sensor detects that a baseball bat, pitcher's arm, orgolf club has passed a prescribed point, or a timing at which the sensordetects that a ball placed on a golf tee or artificial turf mat hasdisappeared is detected as the reference image frame.

[0162] When a change in weight is detected, a frame corresponding tothat detection timing is detected as a reference image frame. A frame ata timing at which the weight of a golf tee or artificial turf mat haschanged is often suited to a reference image frame.

[0163] When the physical switch operates, a frame corresponding to thatoperation timing is detected as a reference image frame. For example, atiming at which a pitcher steps on a pitching plate, a timing at which apressure acts on a golf tee or artificial turf mat, or the like may bedetected.

[0164] The image composition step S4604 executes strobe imagecomposition of the plurality of image frames stored in the image holdingstep S4602 on the basis of the reference frame detected in the referenceimage frame detection step S4603. In this case, the plurality of imageframes stored in the image holding step S4602 may undergo strobecomposition immediately after the reference image frame detection stepS4603 detects a reference image frame. Alternatively, the plurality ofimage frames stored in the image holding step S4602 may undergo strobecomposition when a predetermined period of time elapses after detectionof a reference image frame. The former process is executed when thereference image frame is detected as the last frame of the plurality offrames which are to undergo strobe composition. The latter process isexecuted when the reference image frame is detected as a middle one(e.g., the instance of impact of hitting of baseball or golf swing) ofthe plurality of frames which are to undergo strobe composition. As thenumber of frames which are to undergo strobe composition, all imageframes held in the image holding step S4602 may be used, or apredetermined number of frames, e.g., (2M+1) frames, i.e., M framesbefore and after the reference image frame, may undergo strobecomposition.

[0165] The image output step S4605 outputs a composite image frame thathas undergone the strobe composition process in the image compositionstep S4604 as display data or a video file.

[0166] Detecting the reference image frame serving as a reference for aplurality of frames, which are to undergo strobe image composition, onthe basis of the strobe image composition conditions used to determinean image frame is suited to storage image composition amounts toautomatic designation of a period of the plurality of image frames. Inthis way, a system which can shorten the time required fromphotographing until a strobe composite video is output, and can output astrobe composite image without the intervention of operator's processescan be realized.

[0167] (11th Embodiment) FIG. 37 is a flow chart of a strobe imagecomposition method according to the 11th embodiment of the presentinvention. Referring to FIG. 37, reference numeral 4701 denotes an inputvideo which is input to generate a strobe composite image, and includesa plurality of image frames that represent an object; 4702, a featurepoint tracing step of obtaining the moving locus of a feature point ofan object; 4703, a motion pattern determination step of making motionanalysis of the object on the basis of the obtained locus pattern; and4704, a motion pattern determination result determined in the motionpattern determination step.

[0168] The feature point tracing step 4702 traces a feature point of anobject that appears in the input video 4701. The user may manually tracethe feature point using a pointing device, or the feature point may beautomatically traced using color information of an image. Upon tracing,for example, a histogram intersection method which generates a colordistribution histogram of a rectangular region around the feature point,and searches for regions having similar color distribution histograms inrespective frames can be used. If an input video has already undergonestrobe composition, a plurality of feature points are superposed on asingle image frame, and the feature point can be traced more easily.FIGS. 38A and 38B show the tracing results of a club head and a golfer'shead top in golf swing as examples of the traced feature points. FIG.38B shows an image around the head of an object shown in FIG. 38A in anenlarged scale, and shows the locus of the golfer's head top.

[0169]FIGS. 39A and 39B show the execution state of the motion patterndetermination step. The motion pattern determination step 4703 makesmotion analysis of an object on the basis of the pattern of the featurepoint tracing result as the locus of the feature point obtained in thefeature point tracing step 4702. Note that FIG. 39A shows a case whereinthe motion pattern of the club head is analyzed, and FIG. 39B shows acase wherein the motion pattern of the golfer's head top is analyzed. InFIGS. 39A and 39B, reference numerals 4901 and 4904 denote inputs to themotion pattern determination step 4703, which correspond to the featurepoint tracing results output from the feature point tracing step 4702.Reference numerals 4902 and 4905 denote motion pattern determinationunits each of which determines a motion pattern by comparing the patternof the feature point tracing result with a pattern registered inadvance. Reference numerals 4903 and 4906 denote output examples of themotion analysis results determined by the motion pattern determinationunits 4902 and 4905.

[0170] In the locus pattern of the club head of golf, when a player isviewed from his or her front side, if the swing locus of thefollow-through is larger than that through the downswing, a good,relaxed swing that exploits the weight of the club head is normallydetermined. A club head motion pattern which represents such good swing,that which represents a poor swing, that which represents a swingincident to a beginner, and the like are registered in advance in themotion pattern determination unit 4902. One of the registered motionpatterns, which is most similar to the input 4901 of the motion pattern,is selected, and the motion analysis result 4903 is output. In theexample of FIG. 39A, the motion analysis result 4903 indicating “expert”is obtained based on the input motion pattern 4901.

[0171] As for the locus pattern of the golfer's head top, when a playeris viewed from his or her front side, a pattern which suffers lessdeviations in the horizontal and vertical directions is determined as agood swing in which the upper body does not sway and the backbone axisis stable.

[0172] In an expert's swing, from a state wherein the weight has shiftedto the right foot on the take-back (a state 4907 wherein the locus movesto the left end in FIG. 39B), the weight shifts to the left foot towardimpact at which the golfer hits a ball (the locus moves to the right, asindicated by 4908 in FIG. 39B). Immediately before the impact, a clubhead and arms pull at each other to accelerate the club head (at thistime, the locus of the head top moves to the left in FIG. 39B again),and the impact is reached in a behind-the-ball state 4909. After theimpact, the weight naturally shifts to the left foot (4910). This is themotion pattern of the head top in the expert's swing.

[0173] As in the aforementioned club head, a head top motion patternwhich represents the expert's swing, that which represents a poor swing,that which represents a swing incident to a beginner, and the like areregistered in advance in the motion pattern determination unit 4902. Oneof the registered motion patterns, which is most similar to the input4901 of the motion pattern, is selected, and the motion analysis result4906 is output. In the example of FIG. 39B, the motion analysis result4906 indicating “expert” is obtained based on the input motion pattern4904.

[0174] According to this embodiment described above, in an imagecomposition method that generates a strobe composite image bysuperposing a plurality of frames of a moving image, the moving locus ofa feature point of an object is obtained, and the motion analysis of theobject is made on the basis of the obtained locus pattern. Thus, notonly a strobe composite image is visually output, and a motion analysisresult that can lead to improvement of the athletic ability and the likecan be output together.

[0175] (12th Embodiment) FIG. 40 is a flow chart of a strobe imagecomposition method according to the 12th embodiment of the presentinvention. Referring to FIG. 40, S5001 is an object region extractionstep of extracting an object region from the currently captured imageframe in real time; S5002, a reference frame designation step ofdesignating an image frame which serves as a reference upon executingstrobe composition; and S5003, an image composition step of makingstrobe composition using the designated reference frame and extractedobject region.

[0176] The object region extraction step S5001 always extracts an objectregion in real time irrespective of whether or not an input image is aframe which is to undergo a strobe composition process. The referenceframe designation step S5002 designates an image frame which serves as areference upon executing strobe composition by, e.g., manual designationof the user or the automatic method described in the 10th embodiment.The image composition step S5003 makes strobe composition using thedesignated reference frame and extracted object region. In the strobeimage composition methods described so far in the above embodiments,after an image frame serving as a reference upon executing strobecomposition is designated, object regions are extracted from a pluralityof frames which are to undergo strobe composition, and strobecomposition is made using these object regions. However, such methodsuffers a problem that the object region extraction processing time isadded to the time required from determination of an output video rangeuntil output of an actual video. However, as described in thisembodiment, when the object region extraction process is always made inreal time, this problem can be solved.

[0177] In a strobe image composition method that generates a strobecomposite image by superposing a plurality of frames of a moving image,the control waits for designation of an image frame serving as areference upon executing strobe composition, while an object region isextracted from the currently captured image frame in real time. Afterthe reference image frame is designated, strobe composition is madeusing this designated reference image frame, and the extracted objectregions. Hence, the time required from photographing until output of astrobe composite video can be greatly shortened.

[0178] Additional advantages and modifications will readily occur tothose skilled in the art. Therefore, the invention in its broaderaspects is not limited to the specific details and representativeembodiments shown and described herein. Accordingly, variousmodifications may be made without departing from the spirit or scope ofthe general inventive concept as defined by the appended claims andtheir equivalents.

What is claimed is:
 1. An image composition method for generating astrobe composite image from a plurality of frames of a moving image, themethod comprising: selecting a first frame from the plurality of framesof the moving image; determining a plurality of second frames relatingto the first frame; setting a superposing manner for the strobecomposite image; and generating the strobe composite image bysuperposing the plurality of second frames in accordance with the setsuperposing manner.
 2. A method according to claim 1, whereindetermining the plurality of second frames includes determining thesecond frames based on a reference frame whose time corresponds to thetime obtained by shifting the time of the first frame a certain time. 3.A method according to claim 1, wherein selecting the first frameincludes selecting a frame corresponding to one of start and end framesupon superposing the plurality of second frames.
 4. A method accordingto claim 1, further comprising: selecting a third frame from the movingimage, and wherein determining the plurality of second frames includesdetermining the second frames based on the time of the first frame, andthe time of the third frame.
 5. A method according to claim 1, whereinselecting the first frame includes: selecting a frame of interest inwhich a user is interested; displaying the frame of interest and theframe of the near the frame of interest; and determining the frame ofinterest as the first frame.
 6. A method according to claim 1, whereinsetting the superposing manner includes setting a manner of superposinga frame at a later time on a frame at an earlier time in turn, or amanner of superposing a frame at an earlier time on a frame at a latertime in turn.
 7. A method according to claim 6, further comprising:recording setup information of the superposing manner.
 8. A methodaccording to claim 7, further comprising: generating another strobecomposite image by applying the setup information to another movingimage.
 9. A method according to claim 1, further comprising: displayingthe strobe composite image; designating one of the plurality of secondframes as a designated frame; and changing a superposing order of thedesignated frame to an order different from a superposing order beforedesignation.
 10. An image composition method comprising: inputting amoving image; holding latest N frames (N is a natural number) of themoving image; accepting a one-click-instruction from a user; andgenerating a strobe composite image by superposing the latest N framesin response to the one-click-instruction.
 11. An image compositionmethod comprising: inputting a moving image; holding latest N frames (Nis a natural number) of the moving image; detecting from the latest Nframes a feature frame that conforms to a strobe image compositioncondition; and generating a strobe composite image by superposing thelatest N frames when the feature frame is detected.
 12. A methodaccording to claim 11, further comprising: making a user select one of aplurality of image composition parameter values which can be used forgeneration of the strobe composite image.
 13. A method according toclaim 11, further comprising: generating a plurality of strobe compositeimages by respectively applying a plurality of image compositionparameter values which can be used for generation of the strobecomposite images; and displaying the plurality of strobe compositeimages and making a user select one of the strobe composite images. 14.An image composition method comprising: inputting a first moving image;inputting a second moving image; inputting answer object regions forrespective frames of the first moving image; extracting a plurality ofanswer object images from the respective frames of the first movingimage using the answer object regions; generating an answer strobecomposite image in which the plurality of answer object images aresuperposed; determining an extraction parameter which depends on theanswer strobe composite image; extracting object images from respectiveframes of the second moving image using the extraction parameter; andgenerating a strobe composite image in which the object images extractedfrom the respective frames of the second moving image are superposed.15. A method according to claim 14, wherein determining the extractionparameter comprises: (a) detecting temporary object regions from therespective frames of the first moving image using a temporary extractionparameter; (b) extracting a plurality of temporary object images fromthe respective frames of the first moving image using the temporaryobject regions; (c) generating a temporary strobe composite image inwhich the plurality of temporary object images are superposed; (d)calculating an error between the answer strobe composite image and thetemporary strobe composite image; and repeating (a) to (d) whilechanging the temporary extraction parameter, and determining thetemporary extraction parameter which minimizes the error as theextraction parameter.
 16. An image composition method for generating astrobe composite image by superposing a plurality of frames of a movingimage, the method comprising: displaying respective frames of the movingimage sequentially; selecting a reference frame from the respectiveframes displayed; determining a plurality of frames to be subjected tostrobe composition based on the reference frame; and generating a strobecomposite image by superposing the plurality of determined frames.
 17. Amethod according to claim 16, further comprising: setting a timeinterval, between the respective frames, for displaying the respectiveframes.
 18. A method according to claim 16, further comprising:determining a switching frame at which a superposing manner is switched,and wherein generating the strobe composition image includes switchingthe superposing manner between an overlay manner and an underlay mannerbefore and after the switching frame.
 19. An image composition methodcomprising: determining frames corresponding to start points of strobecomposition; generating strobe composite images by superposing frames inturn based on each of the start points; and displaying the strobecomposite images sequentially.
 20. A method according to claim 19,further comprising: setting a time interval for sequentially displayingthe strobe composite images.
 21. A method according to claim 19, furthercomprising: determining a switching frame at which a superposing manneris switched, and wherein generating the strobe composite images includesswitching the superposing manner between an overlay manner and anunderlay manner before and after the switching frame.
 22. A methodaccording to claim 19, further comprising: determining a plurality offrames to be subjected to strobe composition.
 23. An image compositionmethod for generating a strobe composite image by superposing aplurality of frames of a moving image, the method comprising: inputtinga feature point of an object; obtaining a locus pattern by tracing thefeature point in the strobe composite image; and analyzing a motionpattern of the object on the basis of the obtained locus pattern.
 24. Animage composition method for generating a strobe composite image bysuperposing a plurality of frames of a moving image, the methodcomprising: extracting an image of an object region from a currentlycaptured frame in real time; designating a reference frame for thestrobe composite image; and generating the strobe composite image bysuperposing the image of the object region on the reference frame. 25.An image composition apparatus for generating a strobe composite imagefrom a plurality of frames of a moving image, the apparatus comprising:a selection unit configured to select a first frame from the pluralityof frames of the moving image; a determination unit configured todetermine a plurality of second frames related to the first frame; and asetting unit configured to set a superposing manner used uponsuperposing the plurality of second frames.
 26. An image compositionapparatus, comprising: a generation unit configured to generate a firststrobe composite image by superposing a plurality of frames of a movingimage; a display unit configured to display the first strobe compositeimage; a designation unit configured to designate one of the pluralityof frames as a designated frame; and a change unit configured to changea superposing order of the designated frame to an order different from asuperposing order before designation.
 27. An image composition apparatusfor generating a strobe composite image from a plurality of frames of amoving image, the apparatus comprising: a selection unit configured toselect a first frame from the frames of the moving image; adetermination unit configured to determine a plurality of compositionobjective frames based on the time obtained by shifting the time of thefirst frame a certain time; a setting unit configured to set asuperposing manner of the composition objective frames; a storage unitconfigured to store the superposing manner; and a generation unitconfigured to generate another strobe composite image by applying thesuperposing manner stored in the storage unit to another moving image.28. An image composition apparatus, comprising: an input unit configuredto input a moving image; an image holding unit configured to hold latestN frames (N is a natural number) of the moving image; an accepting unitconfigured to accept a one-click-instruction from a user; and an imagecomposition unit configured to generate a strobe composite image fromthe latest N frames stored in the image holding unit in response to theone-click-instruction.
 29. An image composition apparatus, comprising: aunit configured to input a first moving image and a second moving image;a unit configured to input answer object regions for respective framesof the first moving image; a unit configured to extract a plurality ofanswer object images from the respective frames of the first movingimage using the answer object regions; a unit configured to generate ananswer strobe composite image in which the plurality of answer objectimages are superposed; a unit configured to determine an extractionparameter which depends on the answer strobe composite image; a unitconfigured to extract object regions from respective frames of thesecond moving image using the extraction parameter; and a unitconfigured to generate a strobe composite image in which the objectimages extracted from the respective frames of the second moving imageare superposed.
 30. An image composition apparatus for generating astrobe composite image by superposing a plurality of frames of a movingimage, comprising: a display unit configured to sequentially displayrespective frames of the moving image; a reference frame selection unitconfigured to select a reference frame from the respective framesdisplayed; an objective frame determination unit configured to determinea plurality of frames to be subjected to strobe composition based on thereference frame; and a strobe composition unit configured to generate astrobe composite image by superposing the plurality of frames determinedby the objective frame determination unit.
 31. An image compositionapparatus, comprising: a start point determination unit configured todetermine frames corresponding to start points of strobe compositionfrom a plurality of frames of a moving image; a generation unitconfigured to generate strobe composite images by superposing frames inturn based on each of the start points; and a display unit configured tosequentially display the strobe composite images.
 32. An imagecomposition apparatus for generating a strobe composite image bysuperposing a plurality of frames of a moving image, comprising: aninput unit configured to input a feature point of an object; a featurepoint tracing unit configured to obtain a locus pattern by tracing thefeature point in the strobe composite image; and a motion patternanalysis unit configured to analyze a motion pattern of the object onthe basis of the obtained locus pattern.
 33. An image compositionapparatus for generating a strobe composite image by superposing aplurality of frames of a moving image, comprising: an object regionextraction unit configured to extract an image of an object region froma currently captured frame in real time; a reference frame designationunit configured to designate a reference frame for the strobe compositeimage; and a generation unit configured to generate the strobe compositeimage by superposing the image of the object region on the referenceframe.
 34. A program product comprising a computer usable medium havingcomputer readable program code means for causing a computer to generatea strobe composite image from a plurality of frames of a moving image,the computer readable program code means in the computer program productcomprising: program code means for causing a computer to select a firstframe from the plurality of the moving image; program code means forcausing a computer to determine a plurality of second frames relating tothe first frame; and program code means for causing a computer to set asuperposing manner for the strobe composite image program code means forcausing a computer to generate the strobe composite image by superposingthe plurality of second frames in accordance with the set superposingmanner.
 35. A program product comprising a computer usable medium havingcomputer readable program code means, the computer readable program codemeans in the computer program product comprising: program code means forcausing a computer to generate a first strobe composite image bysuperposing a plurality of frames of a moving image; program code meansfor causing a computer to display the first strobe composite image;program code means for causing a computer to designate at least one ofthe plurality of image frames as a designated frame; and program codemeans for causing a computer to change a superposing order of thedesignated frame to an order different from a superposing order beforedesignation.
 36. A program product comprising a computer usable mediumhaving computer readable program code means for causing a computer togenerate a strobe composite image from a plurality of frames of a movingimage, the computer readable program code means in the computer programproduct comprising: program code means for causing a computer to selecta first frame from the frames of the moving image; program code meansfor causing a computer to determine a plurality of composition objectiveframes based on the time obtained by shifting the time of the firstframe a certain time; program code means for causing a computer to set asuperposing manner of the composition objective frames; program codemeans for causing a computer to store the superposing manner; andprogram code means for causing a computer to generate another strobecomposite image by applying the superposing manner stored in the storageunit to another moving image.
 37. A program product comprising acomputer usable medium having computer readable program code means, thecomputer readable program code means in the computer program productcomprising: program code means for causing a computer to input a movingimage; program code means for causing a computer to hold latest N frames(N is a natural number) of the moving image; program code means forcausing a computer to accept a one-click-instruction from a user; andprogram code means for causing a computer to composite a strobecomposite image by superposing the latest N frames in response to theone-click-instruction.
 38. A program product comprising a computerusable medium having computer readable program code means, the computerreadable program code means in the computer program product comprising:program code means for causing a computer to input a first moving imageand a second moving image; program code means for causing a computer toinput answer object regions for respective frames of the first movingimage; program code means for causing a computer to extract a pluralityof answer object images from the respective frames of the first movingimage using the answer object regions; program code means for causing acomputer to generate an answer strobe composite image in which theplurality of answer object images are superposed; program code means forcausing a computer to determine an extraction parameter which depends onthe answer strobe composite image; program code means for causing acomputer to extract object images from respective frames of the secondmoving image using the extraction parameter; and program code means forcausing a computer to generate a strobe composite image in which theobject images extracted from the respective frames of the second movingimage.
 39. A program product comprising a computer usable medium havingcomputer readable program code means for causing a computer to generatea strobe composite image by superposing a plurality of frames of amoving image, the computer readable program code means in the computerprogram product comprising: program code means for causing a computer tosequentially display respective frames of the moving image; program codemeans for causing a computer to select a reference frame from therespective frames displayed; program code means for causing a computerto determine a plurality of frames to be subjected to strobe compositionbased on the reference frame; and program code means for causing acomputer to generate a strobe composite image by superposing theplurality of determined frames.
 40. A program product comprising acomputer usable medium having computer readable program code means forcausing a computer to generate a plurality of strobe composite images,the computer readable program code means in the computer program productcomprising: program code means for causing a computer to determine aframe corresponding to start points of strobe composition from aplurality of frames of a moving image; program code means for causing acomputer to generate the strobe composite images by superposing framesin turn based on each of the determined start points; and program codemeans for causing a computer to sequentially display the generatedstrobe composite images.
 41. A program product comprising a computerusable medium having computer readable program code means for causing acomputer to generate a strobe composite image by superposing a pluralityof frames of a moving image, the computer readable program code means inthe computer program product comprising: program code means for causinga computer to input a feature point of an object; program code means forcausing a computer to obtain a locus pattern by tracing the featurepoint in the strobe composite image; and program code means for causinga computer to analyze a motion pattern of the object on the basis of theobtained locus pattern.
 42. A program product comprising a computerusable medium having computer readable program code means for causing acomputer to generate a strobe composite image by superposing a pluralityof frames of a moving image, the computer readable program code means inthe computer program product comprising: program code means for causinga computer to extract an image of an object region from a currentlycaptured frame in real time; program code means for causing a computerto designate a reference frame for the strobe composite image; andprogram code means for causing a computer to generate the strobecomposite image by superposing the image of the object region on thereference frame.
 43. A computer system comprising: a processor; a memoryaccessible to the processor; and a software application stored on thememory, wherein the software application comprises: program code meansfor inputting a moving image; program code means for holding latest Nframes (N is a natural number) of the moving image; program code meansfor accepting a one-click-instruction from a user; and program codemeans for generating a strobe composite image by superposing the storedlatest N frames in response to the one-click-instruction.
 44. A computersystem comprising: a processor; a memory accessible to the processor;and a software application stored on the memory, wherein the softwareapplication comprises: program code means for inputting a moving image;program code means for holding latest N frames (N is a natural number)of the moving image; program code means for detecting from the latest Nframes a feature frame that conforms to a strobe image compositioncondition; and program code means for generating a strobe compositeimage by superposing the latest N frames when the feature frame isdetected.
 45. A method according to claim 1, further comprising:displaying the strobe composite image; designating one of the pluralityof second frames as a designated frame; and changing the superposingmanner after the designated frame to a manner different from thesuperposing manner before the designated frame.